feat(observability): full Prometheus metrics endpoint, Grafana dashboard, and indexer-lag alert (Closes #717) - #903

Closes #717

What this fixes

The backend already mounted a /metrics endpoint, but it only exposed five indexer counters. Issue #717 asked for a production-ready Prometheus setup covering:

• request_count and request_duration_ms (HTTP traffic)
• stream_count broken down by status (scheduled, active, paused, completed, canceled)
• claim_count and cancel_count
• indexer_lag_seconds
• Basic auth on /metrics in production
• A Grafana dashboard and an alert when the indexer lags more than 60s

All six metric families were missing, and there was no Grafana/Prometheus configuration anywhere in the repo.

Root cause

Observability was wired only for the indexer loop (eventsIndexedTotal, ledgersScannedTotal, lastIndexedLedger, etc.). Nothing recorded HTTP request traffic, stream/claim/cancel aggregates were computed only by the internal streamMetrics service (not exposed), and there was no definition of "indexer lag" — so no alert could exist. The /metrics route existed (with basic auth via METRICS_AUTH) but served only the indexer counters.

Separately, the backend test suite and type checker could not run at all on main: a merge artifact in streamStore.ts left a truncated transaction-builder block that was a parse error, and the app-level route tests in index.test.ts/cors.test.ts had drifted from the routes they assert on. CI was red for structural reasons before any feature work.

The fix and why

Metrics (backend/src/services/metrics.ts)

• request_count (Counter, labeled by method/route/status) and request_duration_ms (Histogram) are recorded in requestLogger, the single middleware every request passes through. Route labels use the matched route pattern (/api/streams/:id) to keep cardinality bounded.
• stream_count, claim_count, cancel_count are DB-backed gauges refreshed at scrape time with a 60s TTL cache — the same pattern streamMetrics.ts already uses. Deriving them from the DB (computed status + stream_events rows) means claims/cancels are counted exactly once regardless of origin (on-chain indexer event or backend cancelStream), avoiding code-hook drift and double counting.
• indexer_lag_seconds is computed via prom-client's collect() from the wall-clock time of the last successful indexer poll (recorded in indexer.ts on each successful cycle). Because it's computed at scrape time, a completely stalled indexer still produces a growing lag value — which is exactly what the alert needs. It reads 0 until the first successful poll.

Route (backend/src/index.ts) — refreshes the DB-backed gauges inside /metrics (wrapped in try/catch so a scrape still succeeds if the DB is down), keeping the existing METRICS_AUTH basic-auth protection.

Monitoring stack (monitoring/) — self-contained Prometheus + Grafana setup: scrape config that sends METRICS_AUTH credentials, an alert rule indexer_lag_seconds > 60 (with for: 2m), Grafana provisioning with a dashboard covering all six metric families, and a docker-compose.yml + README. METRICS_AUTH is documented in backend/.env.example.

CI unblockers (required so the PR's checks can run at all)

• streamStore.ts: removed the truncated duplicate transaction-build block from a bad merge (parse error that broke the entire backend build and every test importing the module).
• index.test.ts: refreshed the mocked streamStore/auth/db exports and the route-stack invoke helpers (they were picking middleware like readLimiter instead of the real handler), and updated two stale /api/events assertions to match the current route (3-arg countAllEvents, default page size 20).
• cors.test.ts and the indexer/markComplete test mocks: added the new metrics exports so the modules load.

This approach was chosen over alternatives like scattering .inc() calls through every route handler (drift-prone) or a side-car metrics exporter (infrastructure not present in this repo). The change is additive: no auth, payment, or on-chain contract code is touched, and the indexer change is two pure additions.

How it was tested

• New unit tests (services/metrics.test.ts): 100% statement coverage of metrics.ts — metric registration, request counting, DB refresh with TTL cache (including the cache-hit path), lag computation via collect(), and the reset helpers.
• New route tests (metrics.route.test.ts): /metrics returns 200 with all six families present, rejects missing/invalid METRICS_AUTH, and serves valid Prometheus text format.
• Affected suites: 68 tests pass across metrics, metrics.route, requestLogger, cors, index, and the indexer suites (7 pre-existing skips).
• Regressions: compared against base main with git stash — every failing suite on this branch also fails on main (mostly suites that couldn't even load there). This PR fixes 4 suites that were load-broken on main (index.test.ts, cors.test.ts, streamStore.reconcile, webhooks.integration) and adds zero new failures.
• Type check: tsc --noEmit went from 12 parse errors (base) to 25 pre-existing latent errors in unrelated files (migrations, db.ts allowlist exports, config) — none in files touched by this PR. Lint: clean on all changed files.
• Coverage: metrics.ts is 100% lines / 88% branches.

Follow-ups worth filing separately

• db.ts is missing the allowlist functions getAllowedAssets/addAllowedAsset/removeAllowedAsset/searchStreamsFts/syncFtsIndex that index.ts imports — the multi-token allowlist merge ([FEATURE] Add multi-token support (USDC, XLM, and custom SAC tokens) to contract #593) landed incomplete. This breaks assets.test.ts and several tsc errors. The functions need to be implemented and the tests restored.
• Several pre-existing failing suites are unmasked now that the parse error is fixed (validateEnv.test.ts env-var drift, auth.test.ts timestamp/replay drift, contentType.test.ts stale 415 expectations, indexer.gap.test.ts batch-count drift, stats.test.ts module resolution, streamStore.* test drift). They fail identically on main and should be repaired in dedicated PRs.
• Pre-existing tsc errors in backend/src/migrations/* (better-sqlite3 Database namespace) and indexer.ts (rpcServer possibly null, clawback event type) predate this change.